{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading large datasets\n",
    "\n",
    "**Learning Objectives**\n",
    "  - Understand difference between loading data entirely in-memory and loading in batches from disk\n",
    "  - Practice loading a `.csv` file from disk in batches using the `tf.data` module\n",
    " \n",
    "## Introduction\n",
    "\n",
    "In the previous notebook, we read the the whole taxifare .csv files into memory, specifically a Pandas dataframe, before invoking `tf.data.from_tensor_slices` from the tf.data API. We could get away with this because it was a small sample of the dataset, but on the full taxifare dataset this wouldn't be feasible.\n",
    "\n",
    "In this notebook we demonstrate how to read .csv files directly from disk, one batch at a time, using `tf.data.TextLineDataset`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following cell and restart the kernel if needed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import shutil\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.enable_eager_execution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input function reading from CSV\n",
    "\n",
    "We define `read_dataset()` which given a csv file path returns a `tf.data.Dataset` in which each row represents a (features,label) in the Estimator API required format \n",
    "- features: A python dictionary. Each key is a feature column name and its value is the tensor containing the data for that feature\n",
    "- label: A Tensor containing the labels\n",
    "\n",
    "We then invoke `read_dataset()` function from within the `train_input_fn()` and `eval_input_fn()`. The remaining code is as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CSV_COLUMN_NAMES = [\"fare_amount\",\"dayofweek\",\"hourofday\",\"pickuplon\",\"pickuplat\",\"dropofflon\",\"dropofflat\"]\n",
    "CSV_DEFAULTS = [[0.0],[1],[0],[-74.0], [40.0], [-74.0], [40.7]]\n",
    "\n",
    "def parse_row(row):\n",
    "    fields = tf.decode_csv(records = row, record_defaults = CSV_DEFAULTS)\n",
    "    features = dict(zip(CSV_COLUMN_NAMES, fields))\n",
    "    label = features.pop(\"fare_amount\") # remove label from features and store\n",
    "    return features, label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following test to make sure your implementation is correct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a_row = \"0.0,1,0,-74.0,40.0,-74.0,40.7\"\n",
    "features, labels = parse_row(a_row)\n",
    "\n",
    "assert labels.numpy() == 0.0\n",
    "assert features[\"pickuplon\"].numpy() == -74.0\n",
    "print(\"You rock!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use the function `parse_row` we implemented above to \n",
    "implement a `read_dataset` function that\n",
    "- takes as input the path to a csv file\n",
    "- returns a `tf.data.Dataset` object containing the features, labels\n",
    "\n",
    "We can assume that the .csv file has a header, and that your `read_dataset` will skip it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_dataset(csv_path):  \n",
    "    dataset = tf.data.TextLineDataset(filenames = csv_path).skip(count = 1) # skip header\n",
    "    dataset = dataset.map(map_func = parse_row) \n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a test dataset to test our function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile test.csv\n",
    "fare_amount,dayofweek,hourofday,pickuplon,pickuplat,dropofflon,dropofflat\n",
    "28,1,0,-73.0,41.0,-74.0,20.7\n",
    "12.3,1,0,-72.0,44.0,-75.0,40.6\n",
    "10,1,0,-71.0,41.0,-71.0,42.9"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should be able to iterate over what's returned by `read_dataset`. We'll print the `dropofflat` and `fare_amount` for each entry in `./test.csv`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for feature, label in read_dataset(\"./test.csv\"):\n",
    "    print(\"dropofflat:\", feature[\"dropofflat\"].numpy())\n",
    "    print(\"fare_amount:\", label.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following test cell to make sure your function works properly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset= read_dataset(\"./test.csv\")\n",
    "dataset_iterator = dataset.make_one_shot_iterator()\n",
    "features, labels = dataset_iterator.get_next()\n",
    "\n",
    "assert features[\"dayofweek\"].numpy() == 1\n",
    "assert labels.numpy() == 28\n",
    "print(\"You rock!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we can implement a `train_input_fn` function that\n",
    "- takes a input a path to a csv file along with a batch_size\n",
    "- returns a dataset object that shuffle the rows and returns them in batches of `batch_size`\n",
    "\n",
    "We'll reuse the `read_dataset` function you implemented above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_input_fn(csv_path, batch_size = 128):\n",
    "    dataset = read_dataset(csv_path)\n",
    "    dataset = dataset.shuffle(buffer_size = 1000).repeat(count = None).batch(batch_size = batch_size)\n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we implement a `eval_input_fn` simlar to `train_input_fn` you implemented above.\n",
    "The only difference is that this function does not need to shuffle the rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def eval_input_fn(csv_path, batch_size = 128):\n",
    "    dataset = read_dataset(csv_path)\n",
    "    dataset = dataset.batch(batch_size = batch_size)\n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create feature columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The features of our models are the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "FEATURE_NAMES = CSV_COLUMN_NAMES[1:] # all but first column\n",
    "print(FEATURE_NAMES)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the cell below, create a variable `feature_cols` containing a\n",
    "list of the appropriate `tf.feature_column` to be passed to a `tf.estimator`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_cols = [tf.feature_column.numeric_column(key = k) for k in FEATURE_NAMES]\n",
    "print(feature_cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Choose Estimator \n",
    "\n",
    "Next, we create an instance of a `tf.estimator.DNNRegressor` such that\n",
    "- it has two layers of 10 units each\n",
    "- it uses the features defined in the previous exercise\n",
    "- it saves the trained model into the directory `./taxi_trained`\n",
    "- it has a random seed set to 1 for replicability and debugging\n",
    "\n",
    "Note that we can set the random seed by passing a `tf.estimator.RunConfig` object to the `config` parameter of the `tf.estimator`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "OUTDIR = \"taxi_trained\"\n",
    "\n",
    "model = tf.estimator.DNNRegressor(\n",
    "    hidden_units = [10,10], # specify neural architecture\n",
    "    feature_columns = feature_cols, \n",
    "    model_dir = OUTDIR,\n",
    "    config = tf.estimator.RunConfig(tf_random_seed = 1)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the model defined, we can now train the model on our data. In the cell below, we train the model you defined above using the `train_input_function` on `./tazi-train.csv` for 500 steps. How many epochs of our data does this represent?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "tf.logging.set_verbosity(tf.logging.INFO) # so loss is printed during training\n",
    "shutil.rmtree(path = OUTDIR, ignore_errors = True) # start fresh each time\n",
    "\n",
    "model.train(\n",
    "    input_fn = lambda: train_input_fn(csv_path = \"./taxi-train.csv\"),\n",
    "    steps = 500\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate\n",
    "\n",
    "Finally, we'll evaluate the performance of our model on the validation set. We evaluate the model using its `.evaluate` method and\n",
    "the `eval_input_fn` function you implemented above on the `./taxi-valid.csv` dataset. Note, we make sure to extract the `average_loss` for the dictionary returned by `model.evaluate`. It is the RMSE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metrics = model.evaluate(input_fn = lambda: eval_input_fn(csv_path = \"./taxi-valid.csv\"))\n",
    "print(\"RMSE on dataset = {}\".format(metrics[\"average_loss\"]**.5))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Challenge exercise\n",
    "\n",
    "Create a neural network that is capable of finding the volume of a cylinder given the radius of its base (r) and its height (h). Assume that the radius and height of the cylinder are both in the range 0.5 to 2.0. Unlike in the challenge exercise for b_estimator.ipynb, assume that your measurements of r, h and V are all rounded off to the nearest 0.1. Simulate the necessary training dataset. This time, you will need a lot more data to get a good predictor.\n",
    "\n",
    "Hint (highlight to see):\n",
    "<p style='color:white'>\n",
    "Create random values for r and h and compute V. Then, round off r, h and V (i.e., the volume is computed from the true value of r and h; it's only your measurement that is rounded off). Your dataset will consist of the round values of r, h and V. Do this for both the training and evaluation datasets.\n",
    "</p>\n",
    "\n",
    "Now modify the \"noise\" so that instead of just rounding off the value, there is up to a 10% error (uniformly distributed) in the measurement followed by rounding off."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2019 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
